Test Objective
--------------------------------------
This is QA regular release testing cycle for oFono released. Our testing focus would be
new features exploration and regression testing, to ensure major issues for new features
are exposed in timely fashion.
Test Environment
-------------------------------
General Features:
General PC (pre-installed FC12)
- ofono v0.22
- phonesim 1.2
HFP:
Netbook EeePC901
- MeeGo20100424 image
- Sony Ericsson C905, Nokia N95
- ofono v0.22
- Bluez v4.63
GPRS:
- MBM modem
- ofono v0.22
- UNICOM 3G WCDMA network
Modem Emulator:
- test-server tool in ofono
-------------------------------
Test Scope
-------------------------------
The test will do full light functional validation, mainly focus on checking implemented
functionality of ofono v0.22 (voice call, message, cell broadcast, phonebook, sim,
supplementary service, modem management, voice mail, call volume, call history, message
history, GPRS, HFP, USSD) base on phonesim simulator and some hardware devices.
As the dependence of modem, we will do PPP test with gsmdial later.
Test Summary
-------------------------------
During the v0.21 testing, we found the v0.22 release, so switch to v0.22 immediately.
The test result is positive. Most functionalities work well and more stable,
we reported 1 major Bluetooth bug during HFP test, verified 4 bugs, so far, no blocking,
critical issues remain.
Executed total 276 test cases during this round of testing, and among them:
Pass: 266
Fail: 10
New Bugs
-------------------------------
Bug 2782 Bluetooth lost connection after unload loopback module in PA.
Open bugs
-------------------------------
2515 nor S5 command not really work in modem emulator
2517 nor Non at command did not handle properly in modem emulator
2519 nor S3 command can not work normally without default value
8898 enh Spend long time on getting fail report of sms
10061 GSM Compliance issue in sim phonebook
9453 enh [Phonesim] To support UCS2 encode in Cell Broadcast Message
9371 nor Modem interface information didn't be updated after modem was removed
868 nor ussd notification display is incorrect with UCS2 character set
872 maj Can not deactivate all contexts
Fixed bugs
-------------------------------
8185 nor Segmentation fault in ste modem which may caused by call history
Verified Bugs
------------------------------
870 can't cancel/response ussd notification
434 Can't input pin code
435 Can't reset the pin code as the same old number
443 can't change the password by gsm string
Details
The every checkpoint would be one or more test case to cover and result is for
functionality, besides you may find some functional limitation in comments
------------------------------
Features checkpoints test results comments
Call history Missed call pass
Outgoing call pass
Incoming call pass
Duration pass
Date pass
SMS Send/receive pass
Class0, 1 pass
Set service center address pass
More segments pass
Different character set pass
Voice call Dial pass
Accept pass
Reject pass
Hold pass
Retrieve pass
Release pass
ECT pass
Multiparty call Create pass
Hang up pass
Private chat pass
Hold pass
Sim Phonebook Read pass
Different character set pass
Vcard3.0 pass
SS CF pass
CB pass
CW pass
CLIR/CLIP pass
Setting AOC pass
CF/CB pass
GSM string General strings pass
USSD pass
USSD Send pass
Cancel request pass
Respond pass
Different character set fail #868
Modem Basic function pass
Two modems pass
Abnormal quit fail #9371
Voice mail Get/Set number pass
Message Count pass
Notification pass
CPHS information pass
Networking Basic network information pass
Signal indicator pass
Scan pass
Register pass
SIM file Read IMSI pass
Read/set MSISDN pass
Read Emergency number pass
Read SDN number pass
PIN code pass
PUK pass
Cell broadcast Receive a CB message pass
Receive several CB messages pass
Receive more pages CB message pass
Different language pass
Call volume Set/get volume pass
Mute/unmute pass
Message history Send history pass
Receive history pass
Date pass
HFP Voice call pass
Multiparty call pass
Networking pass
Call volume pass
Audio control pass
GPRS Active context pass
Edit profiles pass
Deactivate context pass
Deactivate all context fail #872
Detach pass
Attach pass
Create context pass
Remove context pass
Modem Emualtor Basic AT command pass
Negative fail #2515,#2517,#2519
-------------------------------
Thanks
Zhigang
Show replies by date
Hi Li,
Li, Zhigang wrote:
The test will do full light functional validation, mainly focus on
checking implemented functionality of ofono v0.22
...
Executed total 276 test cases during this round of testing, and among
them:
Pass: 266
Fail: 10
Are these all manual tests or are you using any automated test suite?
Regards
Sjur Brændeland
Hi Sjur
Are these all manual tests or are you using any automated test suite?
There were 70 cases were automation base on phonesim and it was in our AutoDTI test suite.
In additional, there will be an automated test suite for MeeGo project in the future.
Thanks
Zhigang
Regards
Sjur Brændeland
Test Objective
--------------------------------------
This is regular light testing for oFono upstream release. During this round of testing, we
mainly focus on new feature testing to ensure new features work well and regression
testing to ensure no big regression.
Test Environment
-------------------------------
General PC:
pre-installed FC12)
oFono v0.26
Phonesim 1.4
Huawei E160G Modem, 1552 Modem
UNICOM 3G sim card
UNICOM WCDMA network env
Test Coverage
-------------------------------
The testing is based on phonesimv1.4 and covered most of implemented features in v0.26
(Call history, SMS, Voice call, Multiparty call, Sim Phonebook, SS, Setting, GSM string,
USSD, Modem, Voice mail, Networking, SIM file, Cell broadcast, Call volume, Message
history, HFP, GPRS, Modem Emualtor)
"ppp" and "sms delivery report" features testing are also included in
this round of testing.
Test Result Summary
-------------------------------
Total executed total 311 test cases during this round of testing, and among them:
Pass: 308
Fail: 3
Most functionalities work well. we reported 2 new bug and No regression is detected out.
"ppp": it works well. user can setup connection; disconnect the connection.
website access and ftp upload/download work well. Sometimes oFono v0.25 will crash when
disconnecting ppp, but the issue is fixed quickly in ofono v0.26 :-)
"Sms delivery report": most functionalities work well and more stable. User can
enable/disable delivery report, get/set sms bearer, get delivery report after sending sms
successfully. However, rp -error of sms is not tested for it is hard to test it in real
network.
Issue Summary
-------------------------------
New bugs:
Bug 5193 - the attached status did not update after power off GPRS radio
Bug 4081- Cannot get sms delivery status without international number
Main Open bugs:
Bug 8898 - spend long time on getting fail report of sms
Bug 872 - Cannot deactivate all contexts
Bug 5193 - the attached status did not update after power off GPRS radio
Bug 5117- Crash when receiving unsolicited "+CREG: 0"
Bug 389 - 3G modem fail to work after disable/enable the 3G device in ConnMan
Bug 5193 - the attached status did not update after power off GPRS radio
Bug 4081- Cannot get sms delivery status without international number
Bug 4001 - Segfault in ofono_modem_driver_unregister
Bug 3971 - Cannot set Voice No reply timeout value in CF
Bug 2078 - 3G network service sometimes fail to come out after swtich off/on 3G device
Bug 868 - ussd notification display is incorrect with UCS2 character set
Detailed Test Results:
-------------------------------
to make it more readable, we summarized by feature list. :-)
Features checkpoints test results comments
Call history Missed call pass
Outgoing call pass
Incoming call pass
Duration pass
Date pass
SMS Send/receive pass
Class0, 1 pass
Set service center address pass
More segments pass
Different character set pass
Voice call Dial pass
Accept pass
Reject pass
Hold pass
Retrieve pass
Release pass
ECT pass
Multiparty call Create pass
Hang up pass
Private chat pass
Hold pass
Sim Phonebook Read pass
Different character set pass
Vcard3.0 pass
SS CF pass
CB pass
CW pass
CLIR/CLIP pass
Setting AOC pass
CF/CB pass
GSM string General strings pass
USSD pass
USSD Send pass
Cancel request pass
Respond pass
Different character set fail #868
Modem Basic function pass
Two modems pass
Voice mail Get/Set number pass
Message Count pass
Notification pass
CPHS information pass
Networking Basic network information pass
Signal indicator pass
Scan pass
Register pass
SIM file Read IMSI pass
Read/set MSISDN pass
Read Emergency number pass
Read SDN number pass
PIN code pass
PUK pass
Cell broadcast Receive a CB message pass
Receive several CB messages pass
Receive more pages CB message pass
Different language pass
Call volume Set/get volume pass
Mute/unmute pass
Message history Send history pass
Receive history pass
Date pass
HFP Voice call pass
Multiparty call pass
Networking pass
Call volume pass
Audio control pass
GPRS Active context pass
Edit profiles pass
Deactivate context pass
Deactivate all context fail #872
Detach pass
Attach pass
Create context pass
Remove context pass
Modem Emualtor Basic AT command pass
ppp connect/disconnect pass
webpage access pass
ftp upload/download pass
sms delivery report enable/disable delivery report pass
get/Set sms bearer pass
get fail/successful delivery report fail (basically works except for #4081)
Thanks
Elva