Model Parallel in Torch
Hello, We want to use this model on multiple GPU environment. We have Nvidia's DGX station (tesla v100 vram 16GB x 4).
Do you know how to do model parallel in torch for PG-InstructBLIP?
Or do you have any ideas?
Thank you for your help.
I'm not super familiar with multi-GPU inference, but it should be possible. Since PG-InstructBLIP is just a checkpoint of InstructBLIP, I would recommend looking for solutions for loading InstructBLIP in LAVIS.
I think this issue in LAVIS may help: https://github.com/salesforce/LAVIS/issues/321#issuecomment-1600881046
There is also this blog post that may help (but note that we can't directly use the InstructBLIP from huggingface): https://huggingface.co/blog/accelerate-large-models
Thank you for your advice.
I will try this and I will report if I make any progress.
Is there any direct support for multi-gpu
I haven't tried any of the methods suggested in the GitHub issue: https://github.com/salesforce/LAVIS/issues/321#issuecomment-1600881046
I was hoping for a bit direct approach, requring less time :)
In principle, since the LAVIS models are just torch modules, the newest versions of FSDP may be able to automatically handle this. Perhaps first load the model in cpu and then use fully_shard as shown here: https://docs.pytorch.org/tutorials/intermediate/FSDP_tutorial.html