Generative AI has shown promise in assistive technologies ranging from computer-generated captions or phone apps that describe a camera's view, which could be be life-changing for individuals with disabilities. However, developers must involve individuals with disabilities in development of these tools and the privacy and data security policies that govern them to fulfil that promise.