What's the right way to typeset the word bijection (a mathematical term meaning an injective and surjective map or, equivalently, a map having a two-sided inverse) in English?
bijection (i and j are separate glyphs)
bĳection (ĳ is a single glyph)
The same question can be asked about bijective vs. bĳective.
(I'm not asking about Dutch. I'm also not asking what the typesetting programs currently do in English: their authors are sometimes terribly off, and the programs sometimes exhibit unintended behavior. I'm asking what should be the case in English.)
You appear to be mixing up some similar things, whose distinction matters here (and in your other question):
A letter (in this context) is a unit of an orthography. For example, a is a letter of the (standard) English and Dutch orthography; ij is not a letter of the English orthography, but it is regarded as such in the Dutch orthography.
A character is a unit of an encoding, such as Unicode. For example, U+0061 (a) is encodes the Latin small letter a, and U+0133 (ĳ) encodes the Latin small ligature ij.
A glyph is a unit of a font or typeface and what is actually rendered for you to see. For example, many typefaces have a special glyph (ligature) to render fi to avoid collisions (unless intentionally suppressed by the user). Others use a special glyph for ij in the Dutch language (fonts may be language-sensitive). A font may also simply use the glyphs for i and j to render U+0133 (ĳ). None of this is inherently wrong or correct – it’s a technical choice depending on the typeface’s needs.
Consider this example:
You cannot possibly see it, but both examples in the top row use two glyphs to render ij, while the bottom row uses a single glyph.
Now to your question:
To answer your question literally: Using a single glyph to render ij in bijective is okay if the glyph is not distinguishable from a normal combination of the i and j glyph (e.g., the Libertine Regular example above it does not make any difference whatsoever) or if it only exists to avoid weird collisions without visually connecting the i and j. Otherwise (e.g., in Libertine Italic), I would consider this wrong and irritating, in particular since the glyph would make an optical connection, where we have a morpheme boundary linguistically. Mind that this is more a choice of the font designer that should ideally not concern you as a user.
Using U+0133 (ĳ) to encode the English word bijective is clearly wrong, as this would assign a special meaning of the combination of i and j here, which does not exist. This might break searches, screen readers, rendering (if a fallback font is used to render the ĳ and so on. Most ligatures in Unicode are deprecated and exist only for backwards compatibility with previous encodings.
All of this also applies to Dutch by the way, where bijectie is a typical example of a Dutch word where you do not have ij as a letter or ligature.