I don't like the idea of making him a starter. Stick him in the ~90 IP relief role BTS suggested a have him make some two-times-through-the-order starts if need be.
Throw Bolsinger in AAA and try to acquire a Jesse Chavez type to stick into the front end of the bullpen. Sign a few vets off the scrap heap with minor league deals, like they did with Fausto Hernandez and Wade LeBlanc last year, and that makes for some reasonable depth. Heck, I wouldn't be opposed to adding Wade LeBlanc to the pen if he's non-tendered by Pittsburgh.