David Brin has a species who are formed of non-sentient rings, when the rings reach a sort of maximal size and complexity they pile together into an entirely different being who achieves sapience. Through their lifetimes they can change personality, size and things like job duties by adding rings, or removing damaged rings with healthy ones. It is in his FABULOUS Uplift Series. There are other races there of interest, but that is one I found particularly imaginative and interesting.

The ocean from Solaris. As alien as it comes.

Vernor Vinge's "Fire Upon The Deep" has otter-like hive minds of three to twenty bodies that are connected by audio networking, also shrub-like creatures that rely on their wheeled electronic carts to store long-term memories.

Greg Egan's "Dichronauts" have aliens that cannot see north or south because time there has two dimensions (it's complicated). But they have symbiotic creatures in their heads that can perceive north and south with sonar.

I cant remember the book, but there is one with dog like creatures that act as a pack. A single "dog" is not really sentient. Multiple dogs come together to form a sentient pack that acts like an individual. One of the awesome things though, is that if there are too many "dogs" the entity becomes stupid.

The Prime (MorningLightMountain) in Peter F. Hamilton's Commonwealth series.

The weird tri-gendered aliens from “the gods themselves” by Asimov! The emotional, rational and Parental avatars really blew my mind! The creepy Spider aliens from tchaikovsky’s “children of time” also cool!!

This thread is gold.

I’m reading it right now so gotta mention the Moties from The Mote in God’s Eye. Asymmetrical bipeds with high intelligence.

Octavia Butler's aliens in her Xenogensis series

The grendels in the Niven’s Legacy of Heorot series are pretty interesting

The Ariekei from China Miéville’s Embassytown. Their alien-ness creates the primary conflict in the story.

The Tralfamadorians are my favorite. I don't remember the specifics of their biology but their views on existence completely kicked me.

I would argue that not only are Ted Chiang’s Heptapods (Story of your Life, adapted for Arrival) the most creative species, but their simultaneous perception of reality, compared to our linear cause-and-effect perception, made it the most creative sci fi story ever. Upon learning their written language, called Heptapod B, Dr. Banks (Amy Adams) is transformed into someone who knows her entire future. Read or listen to the short story. It changed my life. That’s not an exaggeration. It is that good.

I’ve always enjoyed the variety of aliens in the Uplift universe by David Brin.

I don’t think I’ve seen the aliens from Blindsight mentioned yet. Peter Watts has a good take on truly alien aliens. Oh, and the sense of dread and foreboding that low-key lives rent free in everything that amazing and presumably dreary bastard pens.

Pequeninos from Card's Speaker for the Dead.

Larry Niven, Steven Barnes and Jerry Pournelle: The Legacy of Heorot. The main antagonists/critters called "Grendels" have a life cycle that they authors based on a species of frog, combined with a biochemistry based on supercharged hemoglobin called 'speed' in the novel. The complete planetary ecosystem just feels both alien and real.

It’s not at all clear what the aliens in Peter Watts’s BLINDSIGHT even are.

I love the race in Iain Banks’s EXCESSION known as the Affront.

The Dwellers from The Algebraist

by Iain Banks

The aliens in Asimov’s The Gods Themselves, which is wild because he rarely did aliens.

Niven's Pierson Puppeteers

Baxter's Qax and Silver Ghosts.

Maybe someone’s already mentioned it below, but the Motiles in Pandora’s Star by Hamilton capture purely calculating evolutionary spirit really well. It’s a foreign way of thinking - totally antisocial- but it’s wholly unique and paired with a really interesting alien form that makes for a great villain

their are a few writers that imagined their alien's brains and built their psychology from there and then their societies. Im trying to remember some of the best ones, but a definite leading candidates are the Tines from Fire Upon The Deep - they're not a hive mind, but a collective intelligence compiled of 4-6 individual tines, huddled together and using sound as their medium of telepathy - they use certain sound frequency for communication between "packs", and another set of frequencies to communicate thoughts and memories within the pack. it's really unique and makes you think.

another one which isn't technically science fiction but it is speculative fiction, is Clan of The Cave Bear, where Jean Auel looked at the neanderthal brain structure and speculated what that could mean about their kind of intelligence and out of that invisioned what kind of intelligence these non-human humans would have - and speculated a sentient specie that is very much like humans but still very different. Clan of the Cave Bear has tons of sequels, but the first one - where it's more than just prehistoric adventures but a peek into a different kind of humanity, a different kind of sentient, is one of the best speculative fiction stories I read.

The wormhole aliens from Star Trek DS9 were clever because of their different view of time.

Children of Time series. Nano virus rapidly evolves spiders, octopuses into sentient species that rules terraformed worlds. Alien biology spawning sentient micro organism infection. And that’s only 1.5 books in. Also, Old Man’s War…not only engineered humans but how they fight all the bizarre aliens.

These creatures are only “alien” in a sense…

I read H. G. Wells’ The Time Machine this year for the first time and found the Eloi and Morlocks fascinating, scary and sad, including their relative intelligence and relationship.

Without spoiling much, the descriptions of other creatures towards the end of the story were also disturbing, including how they presented their “intelligence.”

The Gethenians (Winterlings)
Gender: They are ambisexual and neuter for most of the month, adopting either male or female roles during kemmer, their fertile period, with no fixed gender identity.

Appearance: Dark-skinned with black hair, adapted to the cold planet of Gethen (Winter).

Society: Their gender fluidity shapes their culture, politics (like Karhide and Orgoreyn), and relationships, challenging Genly Ai's binary understanding. 

The people of Gethen (Gethenians) have a cultural perspective where truth is not seen as an absolute, objective fact, but rather a matter of perception and imagination. In the neighboring nation of Orgoreyn, they are often seen as "liars" by the bureaucracy because they do not have a rigid, factual-based concept of truth
Within The Left Hand of Darkness, the Gethenians themselves do not universally "abhor lying" in the way an Earth person might understand it; their entire approach to truth and fidelity is culturally different and a major theme of the novel. Their society is more focused on loyalty, personal relationships, and duty to the individual or the community, rather than a rigid adherence to "fact"

------------
The Left Hand of Darkness is a science fiction novel by the American writer Ursula K. Le Guin (October 21, 1929 – January 22, 2018). Published in 1969.
You want to read it. Trust me.

Motiiles from Peter F Hamilton

The Shrike from Hyperion.

The aliens from "Arrival"

For me it’s the alien from the Alien movies. It’s like a wild animal on earth. No scrupules. No conscience. Just survival. And a biology to survive under the harshest conditions.

Both the aliens species in the Expanse. The one on our plane of existence is a hive mind but their biology and culture are presented as a much more alien idea than they normally are and the ones that are the 'villians' of the story are done very well both motivation and the creepy dread they invoke.

The big ol' ones in Jordan Peele's Nope

I always felt Replicants in the Stargate series were the ultimate, inevitable species - biology advances to tech which self replicates and there’s no stopping exponential reproduction.

I don’t think anything comes close to what’s described in Solaris by Stanisław Lem (1961). The alien is basically a living planet ocean and is intelligent in a way that humans just can’t process. It doesn’t talk or have a real physical form. It just pulls stuff out of people’s subconscious and makes it real. Trying to understand how it works is fascinating and deeply unsettling at the same time. 

Highly recommend, it’s one of the few books I’ve read multiple times. 

Mass Effect has some extremely alien species, namely the Hanar, which appear as airborne jellyfish.

For me, it’s the Elder Things from At the Mountains of Madness.

They don’t feel like variations on familiar life at all. Their bodies, symmetry, and biology suggest an evolutionary history that never intersected with ours, and even imagining them feels unstable, as if the mind keeps reaching for reference points that simply are not there.

What elevates them beyond strange anatomy is the glimpse of their inner world. Their art, cities, and recorded history imply a form of intellect that is not just nonhuman, but misaligned with human intuition. The ruins feel less like remnants of a fallen civilization and more like evidence that intelligence itself can organize around principles we are not equipped to grasp.

They stayed with me because they quietly dismantle the assumption that advanced life must resemble anything recognizable, whether physically, culturally, or psychologically.

Rorschach and the Scamblers from Peter Watts' novel Blindsight.

Specifically, a lack of consciousness in an advanced, spacefaring civilization, and it's reaction to human and posthuman culture. This book got me thinking a lot about the nature of consciousness as an individual and a species.

Ender’s game 2 Pequeninos were unexpected.

Read the Uplift War books by David Brin. He gets WILD with the aliens.

I liked the cords and braids in Anvil Of Stars.

Larry Niven multiple species in his Known Space series of interconnected books.

Pierson’s Puppeteers:

“Pierson's Puppeteers are described by Niven as having two forelegs and a single hindleg ending in hoofed feet, and two snake-like heads instead of a humanoid upper body. The heads are small, containing a forked tongue, rubbery lips rimmed with finger-like knobs, and a single eye per head. The Puppeteer brain is housed not in the heads, but in the "thoracic" cavity well protected beneath the mane-covered hump from which the heads emerge.”

The Outsiders: A race of helium II based fragile squid-like creatures who exist in airless, low gravity places and are information brokers (very high$$) to the various interstellar species.

The Kzinti: bipedal feline war-like

There is an extinct race of Slavers which were telepathic organisms that survived by using their powers to enslave other species and there are various artifacts scattered throughout the galaxy from their empire.

Lots of great short stories and novels set in and around this universe that are definitely worth picking up and reading if you like hard science fiction.

Larry Niven's Grogs

They are sentient sessile plants that call small animals to them telepathically and have their dinner jump into their mouths.

Pretty much all the aliens in scavengers reign. Amazing.

I think the best one for me is the aliens that stank to high heaven that are most outside the box. Asimov's Dark Light Years (can I get Cunninghamed?)

The ecoi from Greg Bear's Legacy.

These are continent sized organisms (a handful of them in their world) in a single planet, taking the role of full ecosystems all on themselves.

Despite their size and complexity, they are more like humongous cells, given everything alive there behaves more like an organ or part of the ecoi, made as a biological automata by a central organism for a single purpose.

The giant koalas that have space navies, live in bunkers, and have an absolute system of honor, who share planets and government with psychic monitor lizards, is pretty unique.

The Alfaen empire, We Few

Not necessarily "alien" per se, but I really enjoyed the uplifted, super-sized jumping spiders, the Portiids, in Adrian Tchaikovsky's Children of Time novel.

Mass Effect has some pretty awesome ones if you look away from the counsel races

Hanar and Elcor come to mind functioning off bioluminescent or pheromone communication.

Or the Krogon and Vorcha if you like the impossible to kill type creatures

I'm a fan of some of the ones Ken MacLeod has in his Engines of Light series. One is hyperintelligent extremophile nanobacteria that lives in comets and just wants the galaxy to be quiet so they can talk and think. Another are the Multipliers, which have limbs that divide down to the atomic level and as such they can manipulate matter at that scale.

The Pattern Jugglers in the *Revelation Space series are also interesting.

The creatures that live on the surface of the neutron star in *Dragon's Egg are interesting from a biological and physical perspective, but they behave a bit too human-like.

The plant intelligence in Sue Burke's Semiosis series is interesting, but not as alien as I might have liked.

The Grendels in Legacy of Heorot are kind of cool and interesting.

I suppose the aliens in Blindsight are worth a mention even if they are a bit bland, as is the slime-mold-like alien in Children of Ruin, which is more interesting.

There are more that will resurface in my mind later on, but those are some of the one that come to mind right away.

Actual human beings in Ann Leckie's Ancillary series. The reverse of a hive mind: some of them exist in as a single person many bodies simultaneously, as do her sentient starships.

Just about anything written by C.J. Cherryh. She is amazing at creating alien cultures. She considers their biology, environment, language and more to create very believable, unique and different aliens.

The Scramblers, Peter Watts’ “Blindsight.”

Pierson’s Puppeteers. Yes Niven might not be in fashion but 12-yr-old me was fascinated by the concept of the Puppeteers.

I liked Rocky from Andy Weir book. (Artemisa)

Greeshka from Martin's a song for lya. A parasitic blob that literally consumes you but also your consciousness lives on in it forever in Bliss.

Can't believe no one has mentioned an absolute classic yet: The Martians from Martian Chronicles by Ray Bradbury. Martians are described as being able to change their elemental nature. They can be solid since at one point in the book one gets shot and killed, but in another instance they are described as having a gaseous form. Also their society seems to slightly mirror human society in its hierarchy and social structure within the household, but their tools and utensils are obviously different. For example, when a Martian reads a book, they just touch it and the book sings back to them. A Martian shotgun is loaded with bees instead of buckshot. There are some other details I'm missing but overall I think it was pretty unique.

The triplets from The Gods Themselves by Asimov

Doublers from Eden by Stanislaw lem were great for how bizarre and incomprehensible they are. I’ve seen others mention it but the ocean of Solaris from lems Solaris is also great

Not the most creative but The Primes from the Commonwealth saga really blew me away

Pilot from farscape

Dragon's Egg life evolving on a neutron star

James White's Sector General has a number of these.

The Glimmung in Philip K. Dick's Galactic Pot Healer

There was a cleverly conceived race of aliens in the vintage novel Star Surgeon by Alan E. Nourse. Part of a major mystery in the story. I hate to say more as it would be a spoiler.

Phantoms - Dean Koontz Turns out NOT to be aliens

Constellation Games - Leonard Richardson

First Contact is with aliens who want to catalog everything about us as a way of archiving humanity against entropy and chaos. Some humans are offered the option of becoming 'slow people'.

The Hypotheticals in Robert Charles Wilson's "Spin"

I can't describe them without ruining the story...

Anything from humanity lost or darwin iv

Alan Dean Foster has a number of unusual aliens in his Commonwealth series.

The Thranx insectoids throughout, and the silicon based lifeforms in Prism in particular.

The Shrike or Rocky.

The Angels, from Neon Genesis Evangelion.

The Hippae from Grass (Sheri S. Tepper)

I don't know if it counts as an actual lifeform but even as an honorary mention I'll say the dimensional anomaly from Annihilation (the film anyway)

The Scarens from Farscape, lizard like creatures who artificially increased their intelligence through eating a particular plant and now dominate the galaxy.

Rorschach and the Scramblers in Blingsight are very cool imo

The various alien species in Cherryh’s work. Especially the t’ca & knnn in the Chanur series

We’ve all read about the classic “humanoid with rubber foreheads” or the “bug-like hive mind” aliens, but some authors go absolutely wild with their alien designs and cultures.

It should be noted that "rubber forehead aliens" are mostly a TV thing due to budgetary constraints. Authors have an unlimited special effects budget, are free to do basically whatever they like, and often do.

 And idk. The Qu from All Tomorrows are pretty weird. 🤔

All the different alternate hominids from The Long Earth.

I quite enjoyed Yaphit from The Orville. Making a gelatinous organism that has a romantic plot and not shapeshift into a humanoid.

The creatures who speak in colour bands from the Rama series by Arthur c Clarke. Octospiders I think they’re called.

Also the reed creatures from proxima.

Not enough love for the Pierson’s Puppeteer 👏🏼

It’s gotta be the heptapods from Arrival.

Honestly Farscape is full of them. Scavengers Reign also had fantastic interaction between flora and fawna.

"Moties" from The Mote in Gods Eye and its sequel by Jerry Pournelle and Larry Niven.

The Moties from Pournelle and Niven’s The Mote in God’s Eye. I need to go reread that again. It’s been too long.

The tree-like aliens in Robert Forward’s “Marooned on Eden”. The main body of the alien is essentially a tree which can slowly move its roots to “walk”. It then has 6 “birds” living in nests which are actually a part of the tree’s physiology and its main sense organs (eye and ears). It can program them to fly away according to a specified path and when it returns it reconnects to the tree’s central nervous system and download what it saw. This gives the creature an odd holographic sense where it is aware of how things looked at different times. Likewise it has 6 small animals which it can program to do small tasks and then return and reconnect.

The Devil In The Dark, ST:TOS set a new standard for me for aliens. I’m sure it’s been surpassed many times by now, but compared to the aliens concepts before it came out, the idea of a silicon based life form, which laid eggs that seemed like geological oddities because who would recognize silicon as anything but inert, really opened my eyes to the idea that aliens might be genuinely alien to us.

The Trisolarians in the 3 body problem trilogy were pretty interesting.