If you are looking for a way to integrate oData and custom authentication, I can recommend the following article: Securing OData services using Basic Authentication.
There is one thing that you should notice:
The moment you start using a custom authentication module, it’s important that you configure IIS for Anonymous Access. Otherwise the built-in authentication modules will kick in even when your own custom authentication module is called.
We forgot to change this and it took us some time to figure out why we always got a login dialog although the authentication module authenticated us successfully.