These files were introduced in newer versions of Android (starting from Android Oreo) and are created by the dex2oattool. The idea here is to store DEX code not inside the OAT structure but independently, with some additional metadata to speed up verification. As with OAT, the file format is not documented and changes between different versions of Android. Its description can be found in Android's vdex_file.h source code file.
Apart from this, a new internal ART format called Compact DEX (CDEX) was introduced in Android 9. It aims to reduce storage and RAM usage by compacting various data structures and de-duplicating data blobs in cases where multiple DEX files are present; it may be encountered when working with VDEX files. The corresponding magic header value to recognize them in this case would be cdex. The most up-to-date description can be found in the compact_dex_file.h source code file.