In a saucepan, bring to a boil the butter, salt, milk, and sugar, stirring continuously.
Once the sugar has dissolved, reduce heat to medium and let simmer until mixture reaches 235 degrees, or the “soft ball” stage. Stir occasionally, but don’t scrape down the sides of the pan.
While your fudge mixture is coming to temperature, roughly chop walnuts. Mix the olive oil, sugar, and pumpkin pie spice together and toss to coat walnuts.. Bake at 350º for 6 - 8 minutes.
Once the fudge mixture has reached temperature, remove from heat and pour into a stand mixer. Add powdered sugar and vanilla. Stir on medium low until completely smooth (2 - 3 minutes).
Line an 8 x 8 baking dish with parchment paper and pour fudge into the pan. Sprinkle walnuts evenly over the top and press down slightly.
Let cool on the counter for 15 - 20 minutes, then chill until firm.