It's a personal decision, but I wouldn't hook them ALL up in series. I'm a fan of the "series parallel hybrid." Renogy has a good discussion on the options here:
This page will go into more depth on series, parallel and series-parallel connections. The purpose of this section is to explain why certain connections are utilized, how to set up to your desired connection, as well as going over what is the most beneficial connection to utilize based on your...
uk.renogy.com
They don't mention this, and it's rare anyway, but if you have an entire panel fail a series setup means you have no power. I've read this is possible though I've never heard of anybody who had it happen. The bigger issue is if one panel gets shaded. With a house that should never happen - you design to prevent it. But with an RV that can happen all the time depending on where you park around trees.
So series connections have a pro and a con. The pro is you give your charge controller a higher input voltage, so you have more hours of "charge time" before you drop below its minimum usable threshold. (It also produces half the current for the same output wattage so you can run smaller cables, though I don't know anyone who does that.) The con is partial shading can greatly reduce the output of one leg. That's why I favor hybrid systems on my installations. Currently I'm running 4x 200W panels in a 2x2 setup. I'm planning on adding 2 more panels next year.
About your batteries, this is all depending on your budget, but AGMs also can only produce about half their "rated" power without being damaged. So you can really only safely tap 50Ah out of a 100Ah battery. If you assume you're getting a full charge on them during the day and the system is shutting down because those batteries are "drained" to their safest minimum, then your fridge + 2x inverters are draining 150Ah overnight. That's about 1.8kWh at 12V. That's a little high, my fridge only uses 1.3kWh each day, but it's also designed for RV use and I only have one inverter. But if you have the money to upgrade those batteries, LiFePO4's can give you twice the "usable capacity" over an AGM. Just swapping those could solve your entire problem, it's just super expensive.
I can't say whether they're any good yet, but I'm in the process of swapping my AGM's to Renogy's "Smart" lithium batteries. A big con with others is you can't charge them below freezing. If you're like me and your batteries live in an uninsulated compartment (in my 5th wheel's front storage bay) and you do any cold-weather camping (I do all the time) that's a big con for lithiums. These things have self-heating functions to solve that. (I'm also going to install them in insulated foam boxes.)
Finally, some inverters have huge internal losses. I don't know what brand you have but Will Prowse did a good overview on various inverters here that might let you decide if replacing one or both is worth it:
I'm on the fence about this one for my own system. My inverter is a Magnum CSW1012, which is a rebranded Sensata. It's rated for a no-load draw of 1.2A - I haven't had time to measure to verify this. (Measuring real-world values is very helpful because a lot of ratings are fudged via "optimal conditions" to make them seem more favorable.) Well 1.2A over 24h is just under 30Ah. That's not HUGE but if I have 200Ah of rated capacity that's still 15% of it. Since all of that is "ideal conditions" and conditions are never ideal, my inverter is definitely an important drain in my system. Yours might be much worse. Look for this value on the data sheets for both of them. If Lithium batteries aren't in your future, maybe your budget would allow replacing the two inverters you have with a single, larger one that can run everything you need? If so you can buy something with a lower internal loss and maybe fix your problem in a single change...