Welcome @da3v! You’re a man after my own heart. That’s exactly how I tend to think about things as well. I can’t tell you which path is right for you, but let me tell you about three similar situations I’ve had in my own life and the choices I’ve made:
IFR TONAL NUMBERS: Believe it or not, I experienced some of the same doubts when I was first experimenting with the idea of switching all of my thinking and language over to the tonal numbers instead of the letter names. At first I didn’t even know if it would be viable for me in my own music practice. But that doubt was soon cast aside by the incredible benefits that I was discovering with each passing day. But even then, I still doubted whether anyone else in the world would be willing to think and talk about music in this way. Needless to say, those fears were unfounded. IFR is successful today precisely because we DID have the courage to throw out all of the junk that had cluttered up mainstream music teaching, and start fresh with a much more powerful and adequate language.
TUNING THE GUITAR IN PERFECT FOURTHS: You may not know this about the guitar, but the traditional tuning of the strings is irregular. This prevents you from taking advantage of an incredible benefit that string instruments could offer, which would be a way to visualize any harmonic shape directly on the fretboard. Guitarists still rely heavily on the visualization of shapes, but they have to learn THREE different physical displacements for every interval. For larger structures like scales and chords, this complexity gets multiplied to the point where guitarists don’t even see some of the most obvious and beautiful connections and parallels in western harmony. So the superiority of a perfectly regular tuning was clear. But could I live without being able to play other people’s guitar compositions note for note? And could I live without being able to make use of the entire body of teaching literature for guitar? But much like the IFR Tonal Numbers, from the day I began experimenting with a perfectly regular tuning (all strings separated by perfect fourths), the new discoveries and possibilities were opening up like a waterfall. It’s been 15 years and I’ve never looked back, and I love the perfect fourths tuning more each day. So again, the lesson is clear:
Enormous benefits accrue to those who listen to their inner voice of common sense.
TRUMPET NOTE NAMES: While my main instrument is the guitar, I’m also an amateur trumpet player and I’ve been enjoying the trumpet more and more over the past couple of years. I learned the note names as a kid when I played the trumpet in my high school band. But as an adult musician returning to the trumpet, I knew that I would never have to read trumpet parts. So I considered this very same question that you’re raising. The trumpet is in Bb just like the tenor sax. So why not just learn the fingering for the note “C” and call it a Bb just like any pianist would call that sound? I have not taken that step (yet), but it’s not because there is any danger in doing it. I think it’s a perfectly valid way of developing your own mastery of your instrument, and I don’t see why you can’t give the notes any names you like. But in my own case, the benefit just isn’t quite strong enough. Because I already learned the note names quite deeply as a child, and because I do all of my thinking in IFR Tonal Numbers anyway, I kind of don’t care what key I’m playing in. So at this point, I think that adopting a new naming system would hinder my playing more than helping it.
So this is the final point I wanted to get to. You’re right that you don’t need the traditional note names if you don’t care about reading charts for alto sax. And you’re also right that learning the concert pitch note names would facilitate reading all of the other music literature that is out there. So if you’re just getting started on the alto sax and you have no intention of reading sheet music or exercise books for alto sax, then there’s really no other reason to feel tied to the traditional note names. But I also want to plant the seed of a different idea, which is that once you really understand harmony, you can be equally successful with either set of names.
Here’s a quick example of how this plays out in my case with the traditional trumpet note names. Let’s say I’m going to play with a group, and we’re going to play a piece of music that was composed by one of the other musicians. Imagine that this musician gives me the sheet music in concert pitch. As I scan through that document, in my mind I’m not transposing those notes to Bb trumpet. I’m translating them into tonal numbers. Once I’m able to grasp that entire composition in tonal numbers, I’m quite happy to play it in any key that you like.
Obviously this isn’t practical if the composition is 10 pages long and goes through 50 key changes. (I can’t visualize a structure that large in my mind.) But in my case, I am never in this situation because I would hate to play a composition like that, and I have no interest in playing from sheet music. What I love is improvisation and dialogue among the musicians, and the best vehicles for that experience are much shorter compositions that I can feel and play by ear.
So this is what you have to decide:
Do you care about reading alto sax literature?
Do you care about reading concert pitch literature?
If you’re just starting out then you have no bias for the traditional names. So if you also don’t care about reading books written for alto sax, then it becomes simply a question of which system opens the most doors for you. And since there is about a thousand times more literature written in concert pitch than transposed for alto, the winner of that contest is pretty clear. 
