Ruby key getting replaced, instead of a new key created

ruby 2.5

I have the following code:

test = {'primer' => 'grey'}
layers = ["tan","burgundy"]
fillers = ["blue","yellow"]
layers.each do |l|
    fillers.each do |f|
      test[l] = {} if !test.respond_to?(l)
      test[l][f] = {} if !test[l].respond_to?(f)
    end
end

When I run it in irb, I get the following:

{"primer"=>"grey", "tan"=>{"yellow"=>{}}, "burgundy"=>{"yellow"=>{}}}

I am expecting:

{"primer"=>"grey", "tan"=>{"blue"=>{},"yellow"=>{}}, "burgundy"=>{"blue"=>{},"yellow"=>{}}}

Why does the first respond_to produce the key, when the second one, replaces the previous key?

What am I missing?