any method used to remove well adhered silicone sealant from a surface is going to have trade offs.....
after a lifetime of boat and RV use and maintenance, I can tell you that the easiest way to remove silicone and any of its residue is by using a small, fine wire stainless steel brush on it.....
it doesnt take much effort to break it up when you brush it, and you will never gouge the paint like a metal scraper will, but I must say that if you would use a razor scraper to cut the bigger, thicker areas down a bit, it would help the brushing to go fairly fast, because the brush takes a while to work its way thru blobs, but on the "normal" sized caulk strips, its the best....
I like to use a lubricant (WD-40) as it seems to help the particles break away and fall to the ground as its scrubbed..... the wire brush should be the stainless steel type that aluminum welders use to clean their weld area, which have fine wire bristles... Ive used it on wood, gelcoat and urethane paint without any issues or damage, but the cheaper automotive paints may be a bit softer.... but on my trailer/vehicle or equipment, I would not hesitate to use it simply because it works so well and will look like as if it has never been caulked before, and any marks it may make will be at the caulk line anyway, so it will never be noticed after the new caulk is applied.
but if you arent seeing the results you expect, dont push harder on the brush thinking it will remove it faster.... it wont. but if you brush a little faster, then you will start seeing the action.....