I just understood one interesting thing: the notion of "observable universe" is not limited to light (in the general sense). Gravity is also moving at the speed of light so whatever the masses laying outside of our observable universe, they do not affect us, will not affect us, will never affect us whatever they do.
Even weirder: the notion of "observable universe" is centered on the observer. So imagine a galaxy at the limits of our observable universe, but inside. Imagine it's affected by something lying just outside of OUR observable universe because that "something" is inside ITS observable universe. We'll never see how it is affected either. Because the total time of gravity application from source to galaxy and then of light travel between the galaxy and us is greater than the age of the universe.
In short, it's impossible, for all our observable universe, to observe any effect on anything inside it from outside it, even if that "anything" is closer to us. This drives me crazy, it's another event horizon.